The invention relates to a method of detecting objects in the vicinity of a road vehicle up to a considerable distance, in which the distance from a moving or stationary vehicle to one or more objects is calculated by distance-based image segmentation using stereo image processing, and characteristics of the detected objects are determined by object recognition in the segmented image regions. Image regions of elevated objects and/or flat objects are determined and the elevated objects and/or flat objects are detected by combining (clustering) 3D points in accordance with predetermined criteria. The elevated objects are determined through features with similar distance values and the flat objects are determined through features with similar height values. The relevant objects are followed over time (tracking) and their distance and lateral position relative to the particular vehicle is determined in order to assess the dynamic behavior of the relevant objects.
In order to assist the driver of a motor vehicle in road traffic, driver assistance systems have been developed, which are suitable for detecting situations in the road traffic which are anticipated to be hazardous. Such driver assistance systems can either warn the driver, on the basis of his behavior, or intervene in the management of the vehicle. The intention here is to increase driving safety, to relieve the driver of monotonous driving tasks and, therefore, for driving to become more convenient.
On account of the high requirements on the reliability of systems which increase safety, at the current time, it is predominantly convenience systems which are available on the market. Examples of this are parking aids and intelligent cruise control systems. Driver assistance systems which increase safety are intended to register the surrounding traffic situation to an ever increasing extent and to take it into account.
EP 0 558 027 B1 discloses a device for registering the distance between vehicles. In the case of this device, a pair of image sensors generates an image of an object, which is displayed to the driver. One region of this image is subdivided into windows. The distances from the driving vehicle to the object, which is located in the respective window, are registered. In this case, the distances are calculated by comparing two items of image information recorded by different image sensors in different windows. On the basis of the determined distance information, the respective object is determined. A grid which divides the relevant image region is used. The grid surrounds the object to be registered and supplies further image information. The symmetry of this image information is determined, and the existence of a vehicle travelling in front is predicted by determining a level of stability of a horizontal movement of a line of symmetry and a second level of stability of the distances over time.
This known registration device is used for the purpose of registering and recognizing vehicles located in front of the moving vehicle. The reliable recognition of objects is achieved only in the near region, however, since there the simple registration of lines of symmetry can be carried out with sufficient stability. In the remote region, this simple registration of symmetry is no longer adequate on its own because of the low resolution in the image and the resulting inaccuracy in the determination of the object.
However, high requirements have to be placed on reliable object recognition in particular, in order that the driver is not given any erroneous information, which can lead to erroneous and hazardous reactions. In the case of intelligent systems, the vehicle itself could react in a manner presenting a traffic hazard, on the basis of the erroneous information. Reliable information is imperative, for example in accurate-lane recognition of vehicles at a considerable distance, both in and counter to the actual direction of travel.
For the recognition of interesting patterns, DE 42 11 171 A1 proposes a method which applies the cross relation of small singular extracts from the entire pattern of interest by means of block-by-block progressive image recognition via a trained classification network.
DE 43 08 776 C2 discloses a device for monitoring the outer space around a vehicle, which is travelling over one lane on a road. The lane is defined by extended white lines. By means of image processing, the course of the road is determined by using three-dimensional position information from sections of the white lines. By utilizing the three-dimensional position information from the white lines, the white lines are separated from three-dimensional objects. For each section, the vertical extent of possible objects is determined. As a result, the coordinates for three-dimensional objects of interest, such as motor vehicles, motor cycles or pedestrians, can be defined in the coordinate system of the vehicle. In addition, it is possible to determine which object is concerned.
The procedure described in DE 43 08 776 C2 for monitoring the outer space around a vehicle requires a great deal of computation. It is always necessary to determine the course of the registered region of the road, in order to be able to determine the position of objects in this road course. Since only a limited amount of computing power is available in a motor vehicle, such a monitoring device is ill-suited to practical use. In addition, the known monitoring device is always referred to the presence of white boundary lines, which may not be found on the course of all roads.
EP-A-0 874 331 discloses the practice of dividing up a distance image into regions in the lateral direction away from the vehicle. In this case, a histogram relating to the distance values in the individual regions is drawn up, in order to determine the distances of individual objects from these histograms. The possibility of a collision or contact with objects or other vehicles on the roadway is determined from the position and size of the objects or vehicles. The relative speed of the objects in relation to the particular vehicle is determined by tracking the objects. A reliable statement relating to the relevance of the objects to the situation is possible only after a very computationally intensive procedure, which calls a practical application in road vehicles into question.
The object of the invention is to specify a method of detecting objects in the vicinity of a road vehicle up to a considerable distance which permits the reliable registration of objects, in particular of vehicles in front of and/or behind the road vehicle and their relevance to the situation on the basis of its position relative to the road vehicle.
According to the invention, this object is achieved by determining for the purpose of object recognition, object hypotheses, which are verified by comparison with object models. Segmented image regions are scanned in accordance with predetermined, statistically verified 2D features of the objects to be recognized. The detected objects are compared by using a neural network for the classification of a specific object type. The subclaims relate to advantageous developments of the subject of the invention.
Accordingly, a method of detecting objects in the vicinity of a road vehicle up to a considerable distance is provided, in which the distance from a moving or stationary vehicle to one or more objects is calculated by distance-based image segmentation by means of stereo image processing, and characteristics of the detected objects are determined by object recognition in the segmented image regions.
Determining the characteristics of the detected objects is intended to serve to clarify their relevance to the particular vehicle and therefore contribute to the understanding of the situation.
The detection can preferably be carried out to the front or to the rear and employed, for example, to warn of traffic jams, for distance control from the vehicle in front or for monitoring the rear space. In this case, an important point of view is that the relevance to the situation or the potential hazard of the detected objects is determined from their distance to the particular vehicle and the determined relative speed.
Instead of evaluating pairs of stereo images, which are recorded by a stereo arrangement comprising optical sensors or cameras, in principle, even individually recorded images of different origin can be evaluated in order to determine the distance.
Image regions of elevated objects and/or flat objects are determined. Elevated objects and/or flat objects are detected by combining 3D points in accordance with predetermined criteria. Combining is also designated clustering. In this case, the elevated objects are determined through features with similar distance values and flat objects are determined through features with similar height values. By means of this procedure, objects can be recognized and assessed not only reliably with regard to their distance but also with regard to specific features. Distinguishing between elevated and flat objects is therefore easily possible.
Features of similar distance values and/or similar height are combined in order to form clusters. This distinction between elevated and flat objects is very important for reliable object recognition, for example the recognition of other motor vehicles, and the distinction from road markings. Since appropriately high computing powers can be implemented nowadays in modern motor vehicles, image segmentation of this type by means of distance determination and clustering can be carried out reliably and quickly.
The relevant objects are followed over time and their distance and lateral position relative to the particular vehicle are determined, in order to assess the dynamic behavior of the relevant objects. Only with knowledge of the dynamic behavior of the determined objects are practical reactions of the driver or of the vehicle possible. An “anticipatory” mode of driving is therefore promoted.
Furthermore, by means of this tracking, as it is known, phantom objects which occur sporadically can be suppressed, and the entire recognition performance can be increased. In this way, the number of extracted image regions to be classified in the image can be reduced, if these are checked for their local consistency by means of simple time tracking. By means of tracking the detected objects over time, the object characteristics, such as the distance, relative speed and relative acceleration, can be freed of measurement noise, for example by using a Kalman filter.
For the purpose of object recognition, object hypotheses are determined, which are verified by comparison with object models.
In this way, for the purpose of object recognition, the segmented image regions may be scanned in accordance with predetermined, statistically verified 2D features of the objects to be recognized, and the detected objects may be compared by using a neural network for the classification of a specific object type. In this way, reliable object recognition is carried out.
The detected elevated objects may be, in particular, road vehicles, signposts, bridge columns, lamp posts and so on, whereas the detected flat objects may be, in particular, road markings and boundaries such as curb stones, crash barriers and so on. In this way, for example, the position of a road vehicle on a specific road lane can be determined in a simple way.
In addition, it is advantageous to know the relative position and the relative speed of the detected objects relative to one another and to the moving vehicle, in order to determine the relevance of the detected objects to the situation. To this end, the distance measurement is evaluated, and an accurate road-lane object association is determined.
During the image segmentation, one of the recorded pairs of stereo images can be scanned for significant features of objects to be registered. The spacing of the significant features may then be determined by means of cross-relation by comparing the respective features in a stereo image from the pair of stereo images with the same, corresponding features in the other stereo image from the pair of stereo images, recorded at the same time. The disparities which occur are evaluated.
By determining the spacing of significant features in the pixel range, 3D points in the real world are determined relative to the coordinate system of the measuring device. The information obtained in this way from 3D points is therefore determined from different objects, such as vehicles, road markings, crash barriers, and so on.
In addition to the above-described stereo-based approach, in principle object registration methods based on radar and/or infrared signals in the remote range are also possible.